大规模的视觉预训练在各种下游任务中都表现出了令人印象深刻的进步。现有方法主要是通过图像和文本的全局表示形式的相似性或对图像和文本特征上的高级交叉模式关注来对跨模式对齐进行建模。但是，由于只有全局图像文本对齐信息，因此他们无法明确学习视觉区域和文本短语之间的细粒语义对齐。在本文中，我们介绍了Loupe，这是一种精细的语义一致性视觉语言预训练框架，该框架从新颖的游戏理论互动的角度学习了细粒度的语义对齐。为了有效地计算游戏理论相互作用，我们进一步提出了一种不确定性感知的神经Shapley交互学习模块。实验表明，Loupe在图像文本检索基准测试中实现了最新的。如果没有任何对象级的人类注释和微调，Loupe就可以在对象检测和视觉接地方面实现竞争性能。更重要的是，Loupe从大规模的原始图像文本对学习细粒语义的新方向。

了解人类情绪是智能机器人提供更好的人类机器人相互作用的关键能力。现有作品仅限于修剪视频级别的情感分类，无法找到与情感相对应的时间窗口。在本文中，我们介绍了一项新任务，称为视频中的时间情感本地化（TEL），该任务旨在检测人类的情感并将其相应的时间边界定位在带有校准字幕的未修剪视频中。与时间动作本地化相比，TEL提出了三个独特的挑战：1）情绪的时间动态极为多样； 2）情绪提示都嵌入了外观和复杂的情节中； 3）细粒度的时间注释是复杂且劳动密集型的。为了应对前两个挑战，我们提出了一个新颖的扩张上下文集成网络，该网络与粗细的两流体系结构。粗流通过建模多粒性时间上下文来捕获各种时间动力学。细流通过推理从粗流的多晶格时间上下文之间的依赖性来实现复杂的理解，并将它们自适应地集成到细粒度的视频段特征中。为了应对第三个挑战，我们引入了跨模式共识学习范式，该范式利用了对齐视频和字幕之间的固有语义共识，以实现弱监督的学习。我们为新的测试集提供了3,000个手动注释的时间边界，因此可以对TEL问题进行未来的研究进行定量评估。广泛的实验显示了我们方法对时间情绪定位的有效性。这项工作的存储库位于https://github.com/yyjmjc/temporal-emotion-localization-in-videos。

文本指导的图像编辑模型显示出了显着的结果。但是，还有两个问题。首先，他们采用固定的操纵模块来满足各种编辑要求（例如，更改颜色，纹理更改，内容添加和删除），从而导致编辑过度编辑或不足。其次，他们没有清楚地区分文本要求的和文本 - 略带的部分，从而导致编辑不准确。为了解决这些局限性，我们提出：（i）动态编辑块（DEBLOCK），该块（DEBLOCK）以各种编辑要求动态组成不同的编辑模块。 （ii）一个组成预测变量（COMP-PRED），可根据目标文本和源图像的推断来预测deBlock的组成权重。 （iii）动态文本自适应卷积块（dcblock），该块（dcblock）查询源图像特征，以区分文本需要的零件和文本 - iRrelevant零件。广泛的实验表明，我们的DE-NET可实现出色的性能，并更正确，准确地操纵源图像。代码可在\ url {https://github.com/tobran/de-net}中获得。

培训和评估之间的类别差距被特征为少量学习（FSL）成功的主要障碍之一。在本文中，我们首次凭证识别现实图像中的图像背景，作为课堂上的捷径知识，以适应课堂分类，而是超出FSL中的培训类别。一个小说框架COSOC，旨在通过在训练和评估中提取图像中的图像中的前景对象来解决这个问题而没有任何额外的监督。对电感FSL任务进行的广泛实验表明了我们方法的有效性。

过去一年目睹了将变压器模块应用于视力问题的快速发展。虽然一些研究人员已经证明，基于变压器的模型享有有利的拟合数据能力，但仍然越来越多的证据，表明这些模型尤其在训练数据受到限制时遭受过度拟合。本文通过执行逐步操作来提供实证研究，逐步运输基于变压器的模型到基于卷积的模型。我们在过渡过程中获得的结果为改善视觉识别提供了有用的消息。基于这些观察，我们提出了一个名为VIRFormer的新架构，该体系结构从“视觉友好的变压器”中缩写。具有相同的计算复杂度，在想象集分类精度方面，VISFormer占据了基于变压器的基于卷积的模型，并且当模型复杂性较低或训练集较小时，优势变得更加重要。代码可在https://github.com/danczs/visformer中找到。

Although the performance of person Re-Identification (ReID) has been significantly boosted, many challenging issues in real scenarios have not been fully investigated, e.g., the complex scenes and lighting variations, viewpoint and pose changes, and the large number of identities in a camera network. To facilitate the research towards conquering those issues, this paper contributes a new dataset called MSMT17 with many important features, e.g., 1) the raw videos are taken by an 15-camera network deployed in both indoor and outdoor scenes, 2) the videos cover a long period of time and present complex lighting variations, and 3) it contains currently the largest number of annotated identities, i.e., 4,101 identities and 126,441 bounding boxes. We also observe that, domain gap commonly exists between datasets, which essentially causes severe performance drop when training and testing on different datasets. This results in that available training data cannot be effectively leveraged for new testing domains. To relieve the expensive costs of annotating new training samples, we propose a Person Transfer Generative Adversarial Network (PTGAN) to bridge the domain gap. Comprehensive experiments show that the domain gap could be substantially narrowed-down by the PTGAN.

Masked image modeling (MIM) performs strongly in pre-training large vision Transformers (ViTs). However, small models that are critical for real-world applications cannot or only marginally benefit from this pre-training approach. In this paper, we explore distillation techniques to transfer the success of large MIM-based pre-trained models to smaller ones. We systematically study different options in the distillation framework, including distilling targets, losses, input, network regularization, sequential distillation, etc, revealing that: 1) Distilling token relations is more effective than CLS token- and feature-based distillation; 2) An intermediate layer of the teacher network as target perform better than that using the last layer when the depth of the student mismatches that of the teacher; 3) Weak regularization is preferred; etc. With these findings, we achieve significant fine-tuning accuracy improvements over the scratch MIM pre-training on ImageNet-1K classification, using all the ViT-Tiny, ViT-Small, and ViT-base models, with +4.2%/+2.4%/+1.4% gains, respectively. Our TinyMIM model of base size achieves 52.2 mIoU in AE20K semantic segmentation, which is +4.1 higher than the MAE baseline. Our TinyMIM model of tiny size achieves 79.6% top-1 accuracy on ImageNet-1K image classification, which sets a new record for small vision models of the same size and computation budget. This strong performance suggests an alternative way for developing small vision Transformer models, that is, by exploring better training methods rather than introducing inductive biases into architectures as in most previous works. Code is available at https://github.com/OliverRensu/TinyMIM.

Given the increasingly intricate forms of partial differential equations (PDEs) in physics and related fields, computationally solving PDEs without analytic solutions inevitably suffers from the trade-off between accuracy and efficiency. Recent advances in neural operators, a kind of mesh-independent neural-network-based PDE solvers, have suggested the dawn of overcoming this challenge. In this emerging direction, Koopman neural operator (KNO) is a representative demonstration and outperforms other state-of-the-art alternatives in terms of accuracy and efficiency. Here we present KoopmanLab, a self-contained and user-friendly PyTorch module of the Koopman neural operator family for solving partial differential equations. Beyond the original version of KNO, we develop multiple new variants of KNO based on different neural network architectures to improve the general applicability of our module. These variants are validated by mesh-independent and long-term prediction experiments implemented on representative PDEs (e.g., the Navier-Stokes equation and the Bateman-Burgers equation) and ERA5 (i.e., one of the largest high-resolution data sets of global-scale climate fields). These demonstrations suggest the potential of KoopmanLab to be considered in diverse applications of partial differential equations.

In this chapter, we review and discuss the transformation of AI technology in HCI/UX work and assess how AI technology will change how we do the work. We first discuss how AI can be used to enhance the result of user research and design evaluation. We then discuss how AI technology can be used to enhance HCI/UX design. Finally, we discuss how AI-enabled capabilities can improve UX when users interact with computing systems, applications, and services.

Adversarial robustness assessment for video recognition models has raised concerns owing to their wide applications on safety-critical tasks. Compared with images, videos have much high dimension, which brings huge computational costs when generating adversarial videos. This is especially serious for the query-based black-box attacks where gradient estimation for the threat models is usually utilized, and high dimensions will lead to a large number of queries. To mitigate this issue, we propose to simultaneously eliminate the temporal and spatial redundancy within the video to achieve an effective and efficient gradient estimation on the reduced searching space, and thus query number could decrease. To implement this idea, we design the novel Adversarial spatial-temporal Focus (AstFocus) attack on videos, which performs attacks on the simultaneously focused key frames and key regions from the inter-frames and intra-frames in the video. AstFocus attack is based on the cooperative Multi-Agent Reinforcement Learning (MARL) framework. One agent is responsible for selecting key frames, and another agent is responsible for selecting key regions. These two agents are jointly trained by the common rewards received from the black-box threat models to perform a cooperative prediction. By continuously querying, the reduced searching space composed of key frames and key regions is becoming precise, and the whole query number becomes less than that on the original video. Extensive experiments on four mainstream video recognition models and three widely used action recognition datasets demonstrate that the proposed AstFocus attack outperforms the SOTA methods, which is prevenient in fooling rate, query number, time, and perturbation magnitude at the same.